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This  pilot  study  identifies  critical  success  factors  common  to  government 
acquisition  programs.  It  demonstrates  that  how  these  factors  are  identified 
and  used,  how  they  are  measured,  and  how  they  are  influenced  should  be  a 
mandatory  component  of  education  for  every  federal  government  program 
manager  or  program  manager  selectee. 


An  objective  of  program  manage¬ 
ment,  and  of  program  management 
education,  is  repeatable  success  as 
a  program  manager.  It  does  little  good  if 
managers  are  considered  successful  but  do 
not  know  why  they  were  successful  and 
do  not  know  how  to  repeat  their  successes. 
Success  that  is  the  result  of  luck  is  not  re¬ 
ally  success. 

The  use  of  critical  success  factors 
(CSFs)  in  the  management  of  corpora¬ 
tions  has  been  the  subject  of  several 
published  studies.  The  research  we  de¬ 
scribe  here  investigated  CSFs  as  they  ap¬ 
ply  to  acquisition  programs  within  the  De¬ 
partment  of  Defense  (DoD).  The  use  of 
CSFs  in  the  development  of  critical  man¬ 
agement  information  systems  for  the  DoD 
program  manager  would  have  significant 
benefit. 


Although  profit-driven  private  sector 
companies  have  virtual  autonomy  in  their 
selection  of  suppliers  and  partners,  re¬ 
searchers  investigating  CSFs  had  largely 
ignored  federal  government  projects  at  the 
time  this  study  was  done.  Since  then,  a 
few  federal  agencies  have  reported  CSFs 
for  their  organizations,  but  none  of  the 
reports  published  have  indicated  how  the 
data  was  captured  or  validated,  nor  have 
they  provided  a  CSF-based  measurement 
process.  The  activity  in  this  area  within  the 
federal  departments  therefore  seems  to  be 
at  a  fairly  preliminary  stage  of  development. 

The  questions  we  sought  to  answer  in 
this  pilot  study  are: 

•  Are  there  any  general  CSFs  for  DoD 
programs? 
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•  Are  there  any  CSF-related  measures, 
quantitative  or  qualitative,  that  can  be 
used  on  DoD  acquisition  programs? 

•  Are  there  any  significant  differences  in 
the  CSF  data  between  weapon  systems 
and  automated  information  systems 
(AIS)? 

The  Initial  Research  Environment 


The  importance  of  CSFs  in  manage¬ 
ment  first  gained  widespread  attention 
following  publication  of  an  article  by  J.  F. 
Rockart  (1979).  It  showed  the  need  among 
top  executives  for  certain  critical  elements 
of  information,  notg  provided  by  the  man¬ 
agement  information  systems  (MIS)  or  the 
data  analysis  systems  available.  Rockart 
defined  CSFs  as: 

. .  .the  limited  number  of  areas  in 
which  results,  if  they  are  satisfac¬ 
tory,  will  ensure  successful  com¬ 
petitive  performance  for  the  orga¬ 
nization.  They  are  the  few  key 
areas  where  things  must  go  right 
for  the  business  to  flourish.  If  re¬ 
sults  in  these  areas  are  not  ad¬ 
equate,  the  organization’s  efforts 
for  the  period  will  be  less  than 
desired. 


Fie  further  described  them  as  “areas  of 
activity  that  should  receive  constant  and 
careful  attention  from  management.” 

Rockart  showed  that  executives  suf¬ 
fered  from  data  overload,  but  were  starved 
for  the  right  kind  of  data  essential  to  mak¬ 
ing  the  decisions  necessary  to  manage 
their  enterprises  effectively.  This  includes 
the  identification  of  CSFs  as  well  as  the 
establishment  of  indicators  that  can  alert 
the  executive  when  a  CSF  is  changing  or 
when  the  assumption  upon  which  a  CSF 
is  based  is  no  longer  valid. 

The  initial  Rockart  paper  was  closely 
followed  by  the  publication  of  a  method¬ 
ology  for  CSF  identification  developed  by 
Bullen  and  Rockart  (1981).  The  research 
conducted  since  then  has  been  done  either 
through  the  interview  process  as  described 
by  Bullen  and  Rockart,  or  by  the  ques¬ 
tionnaire  method. 


Optical  Success  Factors 


Identifying  and  managing  CSFs,  and 
tracking  them  separately  from  the  ever- 
increasing  amount  of  data  to  which  execu¬ 
tives  are  subjected,  has  been  the  focus  of 
significant  private  sector  research.  Some 
of  the  research  has  limited  the  study  to 
those  activities  over  which  the  program 
manager  has  direct  control  (Cleland  and 
King,  1988);  the  majority  of  researchers 
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have  broadened  the  focus  to  include  ele¬ 
ments  beyond  the  direct  control  of  a 
project  manager,  but  still  within  the  sphere 
of  things  that  either  he  could  manage,  or 
that  could  exert  significant  influence  on 
his  activities. 

Bullen  (Bullen  and  Rockart,  1981)  has 
suggested  that  CSF  identification  be  fo¬ 
cused  on  whether  CSFs  fall  into  one  or 
more  of  several  key  areas.  These  key  ar¬ 
eas,  plus  one  (modification  management) 
we  have  added,  are: 

•  Global  or  industry  related:  These  are 
activities  essential  to  project  success 
that  would  be  true  of  any  project  or 
company  operating  in  the  particular 
environment  (industry  or  business 
area). 

•  External  influences:  These  CSFs  are 
governed  by  external  factors  that  can 
significantly  influence  the  success  of 
your  endeavor. 

•  Internal  influences:  These  are  deter¬ 
mined  by  internal  factors  that  can  sig¬ 
nificantly  influence  project  success. 

•  Current  and  future:  Included  here  are 
time-driven  CSFs  that  are  essential  to 
project  success.  Current  CSFs  are  ac¬ 
tivities  that  must  be  done  in  the  near 
future.  Future  CSFs  are  those  which  are 
long  range.  Planning  for  the  success  of 
future  CSFs  may  be  an  activity  that  re¬ 
quires  immediate  attention. 

•  Temporal  and  enduring:  These  are  sig¬ 
nificant  influences  that  either  have  a 
short-term  duration  or  are  present 
through  most  or  all  of  a  project. 


•  Risk  abatement:  Some  activities  are 
necessary  in  order  to  avoid  significant 
identified  risks  to  project  success. 

•  Performance:  These  are  identifiable 
levels  of  performance  or  achievement 
that  must  be  realized  for  the  project  to 
be  successful. 

•  Special  monitoring:  These  activities  or 
events  require  special  monitoring,  pro¬ 
tection,  or  contingency  planning  in  or¬ 
der  to  assure  project  success. 

•  Quality:  Quality  requirements,  if  not 
met,  will  mean  the  failure  of  the 
project. 

•  Modification  management:  Some  ac¬ 
tivities  or  conditions  that  currently  ex¬ 
ist  or  are  currently  planned  will,  if  not 
changed,  cause  the  project  to  fail. 

Review  of  the  Literature 


Most  research  has  been  focused  on  the 
identification  of  CSFs  for  executive  level 
managers  in  specific  industries,  or  heads 
of  specific  kinds  of  departments,  princi¬ 
pally  MIS  departments.  There  has  also 
been  some  minimal  research  focused  on 
the  diversity  of  applications  of  CSFs.  One 
fairly  common  problem  with  much  of  the 
reported  research  is  that  many  of  the  iden¬ 
tified  CSFs  have  not  been  stated  in  the 
form  of  an  activity,  as  was  clear  in  the 
original  group  of  definitions  given  by 
Rockart  and  noted  above.  This  led  to  the 
identification  of  CSFs  that  were  ambigu¬ 
ous  and  hard  to  measure. 

One  of  the  early  research  studies  that 
demonstrated  this  problem  was  conducted 
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by  Boynton  and  Zmud  (1984).  This  re¬ 
search  focused  on  the  use  of  CSF,  and 
showed  that  CSF  analysis  can  be  used  suc¬ 
cessfully  to  identify  key  concerns  of  se¬ 
nior  MIS  management,  can  be  used  in  de¬ 
veloping  strategic  plans,  and  can  help 
identify  critical  implementation  issues. 
CSFs  can  also  be  used  to  help  managers 
achieve  high  performance  and  establish 
guidelines  for  monitoring  a  corporation’s 
activities. 

Boynton  and  Zmud  also  noted  that  CSF 
analysis  demonstrated  certain  weaknesses. 

They  found  that 
CSFs  were  dif¬ 
ficult  to  use  un¬ 
less  analysts 
possess  the  ca¬ 
pability  to  suc¬ 
cessfully  apply 
an  identifica¬ 
tion  process. 
Some  analyst  or 
manager  bias 
may  be  intro¬ 
duced  through 
the  interview 
process,  and  if 
used  as  a  re¬ 
quirements  analysis  methodology  the  re¬ 
sulting  information  model  may  not  accu¬ 
rately  represent  the  deployment  environ¬ 
ment.  But  the  researchers  concluded  that, 
despite  these  criticisms,  the  CSF  method 
generates  user  acceptance  among  senior 
management,  it  works  well  at  the  policy, 
operational,  and  strategic  levels  of  infor¬ 
mation  resource  planning.  It  forms  a 
bridge  between  corporate  strategic  inter¬ 
ests  and  information  systems  (IS)  strate¬ 
gic  planning. 

Boynton  and  Zmud  (1984)  also  found 
that  CSFs  can  help  identify  issues  that 


merit  close  management  attention,  their 
intended  purpose,  and  they  are  useful  for 
requirements  analysis  in  building  concep¬ 
tual  models  of  an  organization  or  a 
manager’s  role.  This  may  not  be  appro¬ 
priate,  however,  where  managers  have 
difficulty  conceptualizing.  Their  data  sup¬ 
ported  the  assertions  that  CSFs  generate 
enthusiasm  from  senior  managers,  im¬ 
prove  user  communications,  and  build 
managerial  support  for  information  tech¬ 
nologies,  and  that  CSFs  were  particularly 
successful  in  defining  organizational  in¬ 
formation  infrastructures.  Their  data 
also  indicated  that  lower-level  manag¬ 
ers  may  have  difficulty  formulating 
meaningful  CSFs  and  specific  informa¬ 
tion  measures.  This  finding  underscores 
the  need  for  specific  training  in  CSF 
identification  and  analysis  processes 
and  the  need  to  describe  these  factors 
in  terms  of  activities. 

Boynton  and  Zmud  (1984)  conclude 
that  the  weaknesses  attributed  to  CSFs  can 
be  overcome  through  careful  application 
of  the  method,  while  CSF  strength  as  a 
structured  design  process  for  eliciting  both 
MIS  plans  and  managerial  information 
needs  is  key  to  its  success. 

In  another  significant  study,  “Variation 
of  Critical  Success  Factors  Over  Stages 
in  the  Project  Life  Cycle”  (Pinto  and 
Prescott,  1988),  the  authors  hypothesized 
a  set  of  CSFs,  and  then  conducted  a  vali¬ 
dation  study  based  on  empirical  evidence. 
The  objective  was  to  identify  a  set  of  CSFs 
for  each  life  cycle  phase  that  were  gen¬ 
eral  rather  than  company-  or  industry-spe¬ 
cific,  and  to  determine  the  relative  impor¬ 
tance  of  the  CSFs  across  life  cycle  phases. 
The  final  set  of  CSFs  were  identified  and 
related  to  the  life  cycles  during  which  they 
were  important  (see  Table  1). 


"This  research 
focused  on  the  use 
of  CSF,  and  showed 
that  CSF  analysis  can 
be  used  successfully 
to  identify  key 
concerns  of  senior 
MIS  management, 
can  be  used  in 
developing  strategic 
plans,  and  can  help 
identify  critical 
implementation 
issues." 
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Several  identified  CSFs  demonstrate 
the  difficulty  of  not  specifying  the  fac¬ 
tor  in  terms  of  an  activity.  It  is  difficult 
to  measure,  and  therefore  difficult  to 
know  whether  an  activity  has  been  done 
well,  when  it  is  specified  in  terms  as 
ambiguous  as  “technical  tasks”  or  “project 
mission.” 

Zahedi  (1987)  developed  an  evaluation 
of  reliability  of  an  information  system  as 
a  measure  of  the  system’s  success  based 
on  CSFs.  This  research  addressed  the  is¬ 
sue  of  the  difference  between  behavioral 
and  perceived  measures  of  IS  effective¬ 
ness  resulting  from  a  lack  of  conceptual 
foundation  to  guide  proper  measurement 
development,  and  the  absence  of  a  rigor¬ 
ous  program  of  measurement  validation. 
It  identified  the  need  to  define  CSFs  and 
identify  how  they  are  interconnected.  This 
was  another  look  at  a  question  similar  to 
that  investigated  by  Pinto  and  Prescott 


(1988),  but  looking  at  the  set  of  CSFs  from 
a  reliability  viewpoint.  In  each  case  the 
CSFs  were  not  treated  as  isolated  objects 
but  rather  activities  that  are  interrelated. 

In  “The  Multiple  Uses  of  CSFs” 
(Leidecker  and  Bruno,  1984),  the  authors 
stress  the  applicability  of  CSFs  for  strate¬ 
gic  planning  and  business  strategy  devel¬ 
opment,  identification  of  threats  and  op¬ 
portunities,  and  identifying  a  criteria  for 
strengths  and  weaknesses  assessment. 

Walsh  and  Kanter  (1988)  stress  the  im¬ 
portance  of  using  the  CSF  identification 
process  to  identify  major  causes  of  project 
failure  and  then  ranking  these  major 
causes  by  relative  value,  so  that  such  prob¬ 
lems  can  be  avoided  in  future  programs. 

One  of  the  few  comparative  studies 
done  (Chung,  1987)  concluded  that  if  the 
inquirer  wants  to  know  what  management 
is,  then  the  process  view  should  be  stud¬ 
ied.  However,  if  one  wants  to  know  why 


Table  1. 

Importance  of  Various  Critical  Success  Factors  in  the  Project  Life  Cycle 


Phase 

CSF 

Concept 

Project  mission 

Client  consultation 

Planning 

Project  mission 

Top  management  support 

Client  acceptance 

Execution 

Project  mission 

Troubleshooting 

Well-defined  schedule  or  plan 

Technical  tasks 

Client  consultation 

Termination 

Project  mission 

Technical  tasks 

Client  consultation 
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selected  organizations  are  successful  in 
highly  competitive  environments,  then 
one  must  study  the  three  critical  success 
factors  of  corporate  strategies,  human  re¬ 
sources,  and  operational  systems.  His  con¬ 
clusion  is  that  the  truly  successful  com¬ 
panies  deal  with  these  three  CSFs  differ¬ 
ently  from  the  way  they  are  treated  in  other 
companies. 

More  recently,  the  research  has  contin¬ 
ued  with  the  same  commercial  emphasis 
as  described  above,  but  applied  to  current 
business  trends.  One  group  studied  criti¬ 
cal  success  factors  as  they  apply  to  estab¬ 
lishing  strategic  alliances  (Rai,  Borah,  and 

Ramaprasad, 
1996).  Afurther 
study  of  CSFs 
in  business  alli¬ 
ances,  this  time 
with  a  process 
focus  in  the  oil 
and  gas  industry,  was  reported  in  the  trade 
press  (“Seven  Critical  Success  Factors,” 
1996). 

CSF  analysis  has  also  been  used  for 
community  improvement.  This  is  closer 
to  the  public  sector  than  most  studies,  and 
is  an  example  of  the  analysis  being  ap¬ 
plied  to  a  fairly  narrow  focus  area 
(VanDeusen,  1996).  The  researcher 
gleaned  six  factors  from  14  community- 
scale  future  search  conferences  con¬ 
ducted  between  1993  and  1995.  These 
CSFs  are  leadership,  scope,  participation, 
structure,  results,  and  strong  conference 
management. 

Note  once  again  the  ambiguity  and  the 
problem  when  CSFs  are  not  specified  in 
terms  of  activities.  It  is  very  difficult  to 
measure  something  like  “structure”  or 
“scope”  or  even  “leadership,”  especially 
when  something  like  leadership  can 


be  defined  and  measured  in  so  many 
different  ways. 

Business  processes  for  new  product 
development  have  not  escaped  the  appli¬ 
cation  of  CSF  analysis.  A  benchmarking 
research  study  of  161  business  units  (Coo¬ 
per  and  Kleinschmidt,  1996)  identified  the 
CSFs  for  new  product  performance  at  the 
business  unit  level.  The  researchers  found 
that  the  CSFs  fell  into  major  categories. 
Two  key  performance  dimensions — prof¬ 
itability  and  impact — were  identified.  Four 
key  drivers  were  identified:  a  high-quality 
new  product  process,  the  new  product  strat¬ 
egy  for  the  business  unit,  resource  avail¬ 
ability,  and  research  and  development 
spending  levels.  Merely  having  a  formal 
new  product  process  had  no  impact. 

CSF  analysis  has  also  been  applied  di¬ 
rectly  to  people,  to  measure  productivity. 
Christine  Bullen,  one  of  the  leaders  in  the 
application  of  CSF  analysis,  completed  a 
research  study  of  knowledge  worker  pro¬ 
ductivity  (Bullen,  1995).  She  found  that 
the  context-specific  nature  of  personal 
productivity  demands  an  understanding  of 
the  processes  by  which  knowledge  work¬ 
ers  achieve  their  goals  and  objectives. 
Once  the  nature  of  personal  productivity 
is  understood,  measurement  becomes  a 
much  simpler  task  and  the  measures  have 
real  meaning. 

These  studies  all  show  how  CSF  analy¬ 
sis  is  applicable  to  a  wide  variety  of  in¬ 
dustries  and  subsets  of  industries.  CSF 
analysis  has  also  been  effectively  applied 
to  individual  process  areas  within  a  cor¬ 
poration,  such  as  strategic  planning  and 
information  technology  implementation, 
although  it  is  not  routinely  found  as  a  part 
of  strategic  management. 

Research  on  the  application  of  CSFs  to 
program  management,  and  in  particular 


" CSF  analysis  has 
also  been  applied 
directly  to  people,  to 
measure  productiv¬ 
ity." 
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military  program  management,  is  lacking. 
Research  on  evaluating  the  true  critical¬ 
ity  of  identified  CSFs  is  lacking  in  any  en¬ 
vironment,  government  or  private  sector. 
There  is  an  implied  assumption  in  much 
of  the  research  to  date  that  managers  are 
relatively  equal  in  their  ability  to  iden¬ 
tify  CSFs  that  truly  are  critical  to  project 
success. 


The  Research  Mfhod 


This  research  was  conducted  using  the 
survey  questionnaire  method.  The  survey 
instrument  was  developed  based  on  the 
CSF  categories  identified  by  Bullen 
(1995),  as  noted  above,  plus  the  one  cat¬ 
egory  added  by  these  researchers.  In  the 
instrument,  the  participants  were  asked  to 
identify  both  CSFs  according  to  the  indi¬ 
cated  categories  as  well  as  their  recom¬ 
mended  associated  measures  for  each 
CSF.  The  instrument  was  mailed  to  two 
groups  of  program  managers;  those  man¬ 
aging  embedded  system  programs  and 
those  managing  automated  information 
systems.  The  returns  were  analyzed  and 
the  data  captured  so  that  responses  could 
be  made  to  the  initial  research  questions. 
Data  was  separated  between  the  two 
groups  of  program  managers,  and  the  re¬ 
sults  were  examined  for  each  group  indi¬ 
vidually  as  well  as  comparatively  so  that 
common  CSFs  and  measures  could  be 
identified.  The  findings  were  evaluated  on 
this  basis. 

Having  selected  the  survey  approach, 
the  set  of  program  managers  to  whom 
the  questionnaire  was  sent  was  obtained 
from  lists  maintained  by  the  Information 
Resources  Management  College  (IRMC) 
of  the  National  Defense  University  in 


Washington,  DC,  and  by  the  Defense  Sys¬ 
tems  Management  College  (DSMC)  at  Ft. 
Belvoir,  VA.  Those  from  the  IRMC  were 
program  man¬ 
agers  for  the 
development  of 
non-weapon 
systems,  and  are 
identified  as 
Group  B.  Those 
from  the  DSMC 
list  were  pro¬ 
gram  managers 
for  the  development  of  systems  that  are 
identified  as  embedded  systems,  typi¬ 
cally  weapon  systems,  and  identified  as 
Group  A. 


Initial  Research  Results 


GF  Identification 

The  data  received  was  broken  down 
into  two  groups.  The  Group  A  universe 
was  73  program  managers.  There  were 
20  Group  A  returns.  The  Group  B  universe 
was  57  program  managers  (14  returns). 

The  returns  were  examined  for  identi¬ 
fication  of  CSFs  and  related  measures  that 
might  be,  or  are  being,  used.  No  one  pro¬ 
gram  manager  reported  more  than  10 
CSFs.  The  Group  A  program  managers 
collectively  reported  37  different  CSFs. 
The  Group  B  program  managers  reported 
29  different  CSFs.  Some  reported  CSFs, 
even  though  they  were  not  stated  in  ex¬ 
actly  the  same  words,  were  similar 
enough  in  wording  that  they  could  be 
combined.  This  distillation  left  us  with  1 8 
CSFs  common  to  both  groups. 

It  should  be  noted  that  a  few  other 
CSFs  were  identified  in  addition  to  those 


" Research  on  the 
application  of  CSFs 
to  program  manage¬ 
ment,  and  in  particu¬ 
lar  military  program 
management,  is 
lacking." 
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reported  here.  These  were  very  specific 
to  the  particular  program  and  are  there¬ 
fore  not  included  in  this  report  because  this 
research  effort  sought  to  determine  if  there 
were  CSFs  that  applied  generally  within 
Group  A  and  Group  B,  and  if  there  were 
any  common  to  both  groups. 

M  EASURES 

The  respondents  were  asked  to  identify 
measures  they  use,  or  thought  they  should 
use,  for  each  CSF  they  identified.  There 
was  no  requirement  to  limit  the  number 

of  measures  for 
each  CSF.  The 
Group  A  re¬ 
spondents  iden¬ 
tified  41  differ¬ 
ent  measures. 
Each  Group  A  respondent  identified  a  to¬ 
tal  of  at  least  6  measures.  The  Group  B 
respondents  identified  a  total  of  29  differ¬ 
ent  measures.  Three  of  the  respondents  for 
Group  B  identified  no  measures.  One  other 
Group  B  respondent  indicated  that  there 
were  too  many  factors  required  to  come 
together  for  success  and  the  only  real  mea¬ 
sure  is  the  ultimate  outcome. 

Sysyem  Size 

Program  size  was  requested  in  terms  of 
dollar  value  of  the  total  procurement.  The 
responses  received  for  Group  A  indicated 
total  program  sizes  from  $45  million  to 
$30  billion.  The  responses  received  from 
Group  B  indicated  program  sizes  from 
$400,000  to  $3  billion. 

Data  Analysis  Process 

The  CSF  identification  data  was  com¬ 
piled  and  ranked  by  priority.  Each  CSF 
was  given  a  number,  and  the  number  of 


times  each  was  identified  was  recorded. 
Those  CSFs  identified  most  frequently 
were  ranked  highest  in  priority.  This  same 
analysis  was  done  for  both  Group  A  and 
Group  B.  Those  CSFs  identified  as  being 
common  to  both  Group  A  and  Group  B 
were  also  ranked  in  terms  of  frequency  of 
response. 

Measures  identified  by  the  respondents 
are  given  in  a  separate  chart.  The  measures 
reported  were  also  examined  for  their  ap¬ 
plicability  to  the  CSF  identified.  They 
were  also  examined  in  terms  of  their  pri¬ 
ority.  If  a  critical  success  factor  was  seen 
as  high  priority,  we  judged  whether  this 
was  reflected  in  the  measures  identified 
for  that  CSF. 


Analysis  and  Findings 


Table  2  presents  the  CSFs  from  the 
Group  A  responses.  A  unique  component 
of  the  Group  Aresponse  for  CSF  Number 
4,  “technically  competent  program  office 
staff,”  is  establishing  system  engineering 
expertise  within  the  program  office.  This 
aspect  of  technical  competency  was  not 
present  in  the  Group  B  responses.  Table  3 
shows  Group  B  CSFs.  Note  that  “stable 
and  adequate  budget”  was  the  most  fre¬ 
quently  cited  factor.  Its  prominence  in 
comparison  to  the  other  factors  was  not 
as  dramatic  as  the  most  prominent  CSF  in 
Group  A.  Of  the  1 8  CSFs  common  to  both 
lists,  there  were  evident  differences  in 
emphasis,  both  in  terms  of  frequency  of 
reporting  between  the  two  groups  as  well 
as  the  subtleties  of  their  content  (Table  4). 


"The  CSF  identifica¬ 
tion  data  was  com¬ 
piled  and  ranked 
by  priority." 
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Table  2. 

Group  A  Critical  Success  Factors  in  Priority  Order 

CSF 

No. 

Times 

Factor 

12 

15 

Continuous  meaningful  visibility  using  measures 

4 

9 

Technically  competent  program  office  staff 

2 

9 

Clearly  defined  and  stable  requirements,  including  interface 

1 

8 

Stable  and  adequate  funding 

3 

5 

Risk  management 

7 

5 

Schedule  management 

15 

5 

Stable,  qualified  industrial  base 

17 

5 

Effective  vertical  and  lateral  communications 

16 

4 

Management  political  influencing  agents 

6 

3 

Stable  and  adequate  personnel  resources 

8 

3 

Cost  management 

9 

3 

User  involvement,  support,  and  acceptance 

10 

3 

Strong  and  structured  quality  control 

11 

2 

Clearly  and  objectively  defined  project  goals 

19 

2 

Development  and  execution  of  program  management  strategic  plan 

22 

2 

Change  management 

5 

1 

Configuration  management  and  control 

13 

1 

Other  agency  support  for  training  and  government-furnished 
equipment  (GFE) 

14 

1 

Adequate  program  office  resources 

18 

1 

Leadership 

20 

1 

Thorough  system  documentation 

21 

1 

Test  and  evaluation  master  plan  approval 

23 

1 

Program  office  teamwork 

24 

1 

Effective  and  timely  decision  making 

25 

1 

Foreign  military  sales 

26 

1 

Measure  and  control  integrated  logistics  support  performance 

27 

1 

Initiation  of  new  projects 
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Table  3. 

Group  B  Critical  Success  Factors  in  Priority  Order 

CSF 

No. 

Times 

CSF 

1 

10 

Stable  and  adequate  budget 

9 

9 

User  involvement  and  support 

12 

9 

Effective  technical  performance  evaluation 

2 

8 

Detailed  requirements  analysis 

4 

8 

Technically  competent  staff 

19 

7 

Top  management  support 

17 

6 

Effective  lateral  and  vertical  communications 

7 

6 

Schedule  management 

10 

6 

Strong  quality  control  program 

6 

5 

Stable  project  staff 

16 

5 

Management  of  political  influencing  agents 

13 

4 

Other  agency  support:  Training  and  GFE 

3 

3 

Risk  management 

20 

3 

Strong  knowledge  of  life  cycle  management 

23 

3 

Incremental  acquisition 

8 

2 

Cost  management 

22 

2 

Common  sense 

11 

2 

Clearly  defined  mission 

14 

2 

Adequate  program  office  resources 

21 

1 

Objective  economic  analysis 

18 

1 

Leadership 

15 

1 

Stable,  qualified  industrial  base 

5 

1 

Configuration  management 

24 

1 

On-site  team  to  prevent  fraud,  waste,  and  abuse 

Overall  Observations 

Given  the  publicity  generally  afforded 
to  configuration  management,  it  was  a 
surprise  that  this  factor  was  named  only 


once  in  each  group.  This  could  mean  that 
those  reporting  did  not  recognize  the  im¬ 
portance  of  configuration  management,  or 
recognized  its  importance,  but  believe  it 
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Table  4. 

SFsfor  Groups  A  and  B  Combined,  in  Priority  Order 

CSF 

No. 

Times 

CSF 

12 

24 

Continuous  meaningful  visibility  using  measures 

1 

18 

Stable  and  adequate  funding 

18 

2 

Leadership 

2 

17 

Clearly  defined  and  stable  requirements,  including  interface 

4 

17 

Technically  competent  program  office  staff 

9 

12 

User  involvement,  support,  acceptance 

7 

11 

Schedule  management 

17 

11 

Effective  vertical  and  lateral  communications 

10 

9 

Strong  and  structured  quality  control 

16 

9 

Managementof  political  influencing  agents 

3 

8 

Risk  management 

6 

8 

Stable  and  adequate  personnel  resources 

15 

6 

Stable,  qualified  industrial  base 

13 

5 

Other  agency  support  for  training  and  GFE 

8 

5 

Cost  management 

11 

4 

Clearly  and  objectively  defined  project  goals 

14 

3 

Adequate  program  office  resources 

5 

2 

Configuration  management  and  control 

is  done  well  enough  now  not  to  be  a  prime 
candidate  for  program  manager  attention. 

The  questionnaire  asked  the  program 
manager  to  list  activities  that  were  be¬ 
lieved  to  be  critical  to  the  success  of  the 
program.  Issues  reported  will,  in  some 
way,  reflect  those  areas  that  have  required 
a  significant  degree  of  program  manager 
attention. 

Another  unexpected  result  was  the 
placement  in  order  of  prominence  in 
which  risk  management  appeared  in  the 


Group  B  list,  being  number  13  in  order  of 
frequency  of  response.  This  same  CSF  is 
ranked  number  6  in  order  of  importance 
in  Group  A. 

With  regard  to  the  CSFs  common  to 
both  groups,  the  list  reflects  a  strong  be¬ 
lief  that  factor  number  12,  continuous 
meaningful  visibility  using  measures,  is 
of  primary  importance  to  program  success 
for  any  system.  Stable  and  adequate  fund¬ 
ing,  clearly  defined  and  stable  require¬ 
ments,  and  technically  competent  program 
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office  staff  were  the  factors  next  in  order 
of  importance,  and  these  three  are  nearly 
equal  in  prominence.  These  four  CSFs  ap¬ 
peared  more  prominently  than  either  cost 
management  or  schedule  management. 
This  may  reflect  a  belief  that  if  these  top 
four  CSFs  are  accomplished,  cost  manage¬ 
ment  and  schedule  management  are  more 
easily  accomplished.  This  may  also  sug¬ 
gest  that  the  focus  of  the  external  over¬ 
sight  groups — cost  and  schedule — is  not 
among  those  activities  that  are  most  im¬ 
portant  to  program  success,  at  least  not  as 
viewed  by  those  responsible  for  execut¬ 
ing  the  mission  of  program  management. 

One  must  also  recognize  that  the  CSFs 
identified  are  not  necessarily  disjoint.  For 
example,  the  factor  continuous  and  mean¬ 


ingful  visibility  will  be  a  necessary  com¬ 
ponent  of  risk  management.  This  is  like¬ 
wise  true  of  the  CSFs  strong  and  struc¬ 
tured  quality  control,  and  technically  com¬ 
petent  program  office  staff.  All  of  these 
tend  to  be  means  for  managing  program 
risk. 

Measures 


Analysis  of  the  measures  was  accom¬ 
plished  in  light  of  the  identified  CSFs.  It 
was  expected  that  the  most  frequently 
mentioned  CSFs  should  have  measures  re¬ 
flecting  them,  and  those  with  minimal 
mention  may  be  expected  to  have  the  least 
number  of  measures. 


Table  5.  Group  A  Measures  in  Priority  Order 


No. 

Frequency 

Measure 

1 

10 

Stable  and  adequate  budget 

5 

10 

Deviation  from  schedule 

4 

8 

Deviation  from  cost 

2 

7 

N  umber  and  frequency  of  requirements  changes 

6 

7 

Number  of  unique  trouble  reports 

3 

4 

Changes  to  the  budget 

8 

4 

Results  of  tests  of  independent  systems 

20 

4 

Cost  of  change  versus  cost  of  delay  equals  cost  to  improve 

9 

3 

User  or  contractor  walkthroughs  and  reviews 

12 

3 

Funding  level  versus  plan 

7 

2 

Number  of  customer  complaints 

10 

2 

Prime  contractor  productivity  per  7000.2 

16 

2 

Program  plan  assessment  (qualitative) 

21 

2 

Number  of  reworks  or  rewrites 

(continued) 
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Table  5. 

Group  A  Measures  in  Priority  Order  (continued) 

No. 

Frequency 

Measure 

23 

2 

Time  between  problem  occurrence  and  problem  identification 

25 

2 

Time  delay  of  GFE  deliveries 

1 

1 

Quantitative  assessment  of  requirements 

11 

1 

Program  quality  targets 

13 

1 

VROC,  mean  time  between  failures  (MTBF),  Pd,  mean  time 
to  repair  (MTTR) 

14 

1 

System  availability 

15 

1 

Response  to  change  (qualitative) 

17 

1 

Number  of  issues  requiring  higher  approval 

18 

1 

Time  taken  for  approval  decisions 

19 

1 

Number  of  acquisition  protests 

22 

1 

Number  of  first  time  approvals 

24 

1 

Time  to  process  approved  change 

26 

1 

Workload  stability 

27 

1 

Number  and  effect  of  Congressional  interactions 

28 

1 

Effectiveness  of  visibility  processes 

29 

1 

Number  of  delay  or  disruption  claims  from  contractor 

30 

1 

Reject  rates 

31 

1 

Number  of  quality  deficiency  reports 

32 

1 

RAM  measures 

33 

1 

Number  of  miscues  per  month  (no  coordination;  misunderstand) 

34 

1 

Number  of  technical  surprises  per  month 

35 

1 

Number  of  suggestions  adopted  by  contractor 

36 

1 

Number  of  delinquent  action  items— days  late 

37 

1 

Cost  versus  operational  effectiveness 

38 

1 

Number  of  risks  identified,  month 

39 

1 

Number  of  risks  resolved  per  month 

40 

1 

Number  of  qualified  staff  versus  need 

41 

1 

Number  of  physical  resources  versus  need 
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In  Group  A  (Table  5),  the  most  fre¬ 
quently  mentioned  measure  is  number  5, 
deviation  from  schedule.  The  next  most 
frequently  mentioned  measure  is  number 
4,  deviation  from  cost.  This  may  be  re¬ 
flective  of  the  ease  of  data  collection,  and 
the  need  to  respond  to  the  program  execu¬ 
tive  officer,  the  GAO,  Congress,  and  the 
Defense  Acquisition  Board,  rather  than  fo¬ 
cusing  on  those  activities  that  the  program 
managers  clearly  felt  were  of  significantly 
superior  importance  to  program  success. 

Given  the  importance  afforded  the  CSF 
identified  as  continuous  visibility,  a 
number  of  measures  can  be  considered 


reflective  of  this  CSF.  They  are  shown  in 
Table  6.  In  terms  of  sheer  volume,  these 
measures  reflect  the  importance  afforded 
continuous  visibility.  However,  they  were 
distributed  across  the  spectrum  of  those 
reporting,  and  the  majority  of  the  measures 
were  only  identified  once,  an  obvious 
cause  for  concern.  The  two  most  fre¬ 
quently  mentioned  of  all  the  visibility 
measures  are  those  related  to  requirements 
changes  and  those  related  to  trouble  re¬ 
ports.  The  next  most  frequently  mentioned 
measure  was  number  8,  test  results. 

Walkthroughs  and  reviews,  number  9, 
a  widely  publicized  source  of  visibility, 


Table  6. 

Measures  Reflecting  the  I mportance  of  the  CSF  Continuous  Visibility 


No. 

Frequency 

Measure 

2 

7 

Number  and  frequency  of  requirements  changes 

6 

7 

Number  of  unique  trouble  reports 

8 

4 

Test  results 

9 

3 

Walkthroughs  and  reviews 

7 

2 

Number  of  customer  complaints 

10 

2 

Contractor  productivity 

1 

1 

Quantitative  assessment  of  requirements 

11 

1 

Program  quality  targets 

13 

1 

VROC,  MTBF,  Pd,  MTTR 

14 

1 

System  availability 

28 

1 

Effectiveness  of  visibility  process 

31 

1 

Number  of  quality  deficiency  reports 

32 

1 

RAM  requirements 

34 

1 

Number  of  technical  surprises  per  month 

38 

1 

Number  of  risks  identified  per  month 

39 

1 

Number  of  risks  resolved  per  month 
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Table  7.  Group  B  Measures,  in  Priority  Order 


No. 

Frequency 

Measure 

4 

4 

Deviation  from  cost 

5 

4 

Deviation  from  schedule 

6 

4 

Number  of  unique  trouble  reports 

10 

3 

Program  reviews 

12 

3 

Product  quality 

22 

3 

System  throughput  (performance) 

2 

2 

Number  of  system  requirements  changes 

3 

2 

Budget  changes 

13 

2 

Funding  level  versus  plan 

16 

2 

User  feedback 

20 

2 

External  and  internal  independent  verification  and  validation 

7 

2 

Number  of  customer  complaints 

1 

1 

Requirements  review 

8 

1 

User  acceptance 

9 

1 

Test  results  data  reports 

11 

1 

Productivity 

14 

1 

System  reliability 

15 

1 

Downtime:  Rate  and  duration 

17 

1 

Progress  demonstration 

18 

1 

Contractor  product  demonstrations 

19 

1 

Personnel  evaluations 

21 

1 

Milestone  resource  review 

23 

1 

System  backlog 

24 

1 

Analysis  reports 

25 

1 

Evaluation  against  oversight  criteria 

26 

1 

Number  of  support  complaints 

27 

1 

Number  of  software  changes 

28 

1 

Time  to  complete  software  change 
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was  only  mentioned  three  times.  This  is 
cause  for  some  concern,  both  because  of 
what  walkthroughs  and  reviews  can  pro¬ 
vide  that  is  not  being  used,  and  because 
of  the  need  to  assure  that  these  visibility 
mechanisms  are  properly  reflected  in  the 
acquisition  process,  particularly  in  the  re¬ 
quest  for  proposal  and  the  contract.  For 
CSF  number  4,  technically  competent  pro¬ 
gram  office  staff  (the  second  most  fre¬ 
quently  mentioned  CSF),  there  are  virtu¬ 
ally  no  measures  reflected. 

It  is  evident  that  there  is  a  lack  of  cor¬ 
relation  between  those  activities  identified 
as  CSFs  for  Group  A  and  therefore  deemed 

critical  to  pro¬ 
gram  success, 
and  the  mea¬ 
sures  used  or 
suggested.  The 
number  of  mea¬ 
sures  related  to 
the  top  priority 
CSF,  but  that 
were  only  re¬ 
ported  once,  re¬ 
flects  a  lack  of 
consistency 
among  those 
engaged  in  ac¬ 
quisition  activi¬ 
ties  in  terms  of 
how  commonly  recognized  critical  activi¬ 
ties  can  and  should  be  evaluated. 

In  general  for  Group  B  (Table  7),  the 
lack  of  dominance  of  any  one  measure, 
and  the  deviation  of  the  three  measures 
deviation  from  cost,  deviation  from  sched¬ 
ule,  and  number  of  unique  trouble  reports 
being  the  most  frequently  named,  may  be 
reflective  of  a  general  lack  of  familiarity 
with  quantitative  evaluation  processes  as 
well  as  a  lack  of  consistency  among  those 


engaged  in  non-weapon  system  acquisi¬ 
tion  in  terms  of  how  those  activities 
deemed  critical  to  success  can  and  should 
be  evaluated. 

Although  CSF  number  2,  detailed  re¬ 
quirements  analysis,  is  noted  eight  times, 
measure  number  1 ,  requirements  review, 
was  only  mentioned  once,  and  measure 
number  2,  number  of  system  requirements 
changes,  was  only  mentioned  twice.  A  to¬ 
tal  of  16  of  the  28  measures  reported  were 
only  mentioned  once.  Six  of  the  measures 
were  only  mentioned  twice.  Therefore, 
only  6  of  the  28  measures  were  mentioned 
more  than  twice,  although  11  of  the  24 
CSFs  for  Group  B  were  listed  at  least  five 
times.  There  is  a  clear  lack  of  correlation 
between  the  activities  considered  critical 
to  program  success  and  the  management 
information  available  by  which  the  pro¬ 
gram  managers  can  measure,  evaluate,  and 
make  use  of  those  critical  factors.  The 
most  frequently  named  measures  are  more 
reflective  of  response  to  oversight  groups 
than  they  are  of  the  issues  actually  con¬ 
sidered  by  the  program  managers  as  most 
critical  to  program  success. 

One  clear  concern  is  that  if  program 
managers  recognize  certain  activities  to  be 
critical  to  program  success,  and  the  pro¬ 
gram  manager’s  information  network  does 
not  provide  measures  reflective  of  those 
critical  factors,  then  their  ability  to  man¬ 
age  those  factors  is  jeopardized.  The  in¬ 
formation  system  used  by  the  program 
office  will  not  be  sufficiently  supportive 
of  the  program  needs. 

Findings 


Based  on  the  above,  we  can  list  a  num¬ 
ber  of  findings. 


"One  clear  concern 
is  that  if  program 
managers  recognize 
certain  activities  to 
be  critical  to  pro¬ 
gram  success,  and 
the  program  mana¬ 
ger's  information 
network  does  not 
provide  measures 
reflective  of  those 
critical  factors,  then 
their  ability  to  man¬ 
age  those  factors  is 
jeopardized.” 
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•  The  CSFs  for  DoD  program  manage¬ 
ment  are  identifiable,  and  their  explicit 
identification  would  clearly  assist  the 
program  managers  in  maintaining  man¬ 
agement  focus  on  the  factors  most  im¬ 
portant  to  program  success. 

•  A  significant  number  of  CSFs  are  com¬ 
mon  to  both  Group  A  and  Group  B 
types  of  programs. 

•  The  component  assumptions  and  em¬ 
phasis  for  a  given  CSF  common  to  both 
Group  A  and  Group  B  may  be  slightly 
different.  This  difference  is  largely  a 
function  of  the  difference  between  the 
missions  of  the  two  groups,  Group  A 
being  more  concerned  with  the  com¬ 
plete  development  of  total  systems  than 
is  Group  B. 

•  The  CSFs  identified  by  the  program 
managers  as  the  most  significant  for 
program  success  are  not  those  factors 
that  receive  the  most  attention  from  the 
oversight  activities  and  agencies. 

•  The  measures  identified  most  often  by 
the  program  managers  as  those  used  or 
recommended  are  significantly  more 
oriented  toward  cost  and  schedule 
(which  must  be  briefed  to  the  oversight 
agencies)  rather  than  toward  factors 
identified  by  the  program  managers  in 
the  field  to  be  most  critical  to  the  pro¬ 
gram  success. 

•  There  is  no  widely  recognized  and  gen¬ 
erally  used  set  of  measures  consistent 
with  the  most  frequently  reported 
CSFs.  This  leads  to  the  conclusion  that 
even  though  various  factors  are  recog¬ 
nized  as  critical,  they  are  not  usually 


explicitly  identified  and  the  informa¬ 
tion  network  required  to  manage 
against  those  critical  factors  is  not  de¬ 
veloped. 

•  A  commonly  recognized  set  of  CSFs, 
and  a  consistent  measurement-based 
information  network  based  on  these 
CSFs,  would  be  of  significant  benefit 
to  the  program  managers  as  well  as  the 
oversight  agencies.  Such  a  manage¬ 
ment  system  would  significantly  im¬ 
prove  the  management  success  poten¬ 
tial  on  programs  across  the  board,  and 
would  provide  the  external  groups  a 
consistent  way  of  evaluating  and  com¬ 
paring  different  programs  so  that  rec¬ 
ommendations  for  future  improve¬ 
ments  could  be  intelligently  based. 

•  A  CSF-based  information  network  for 
program  management  would  lend  itself 
to  not  only  increased  visibility  and 
awareness  for  the  program  manager 
and  staff  during  all  life  cycle  phases, 
but  would  provide  the  base  for  the  es¬ 
tablishment  of  measures  for  determin¬ 
ing  when  the  underlying  assumptions 
for  a  given  CSF  may  be  changing. 

•  A  CSF-based  information  network 
would  provide  a  common  framework 
for  productive  discussions  between  the 
program  manager  and  the  external 
groups,  including  the  development 
contractors,  and  would  greatly  support 
the  effectiveness  of  IPTs. 

•  A  CSF-based  information  network 
would  significantly  reduce  the  dupli¬ 
cative  reporting  and  diversions  the  pro¬ 
gram  managers  experience  under  the 
present  conditions. 
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•  A  CSF-based  analysis  process  would  not  respond  at  all,  to  have  a  subordinate 
be  a  significant  teaching  instrument  for  provide  the  responses,  or  to  respond  hast- 
educating  prospective  program  manag-  ily  without  giving  the  matter  the  intellec- 
ers  in  strategic  thinking  in  terms  of  tual  time  it  requires, 
those  considerations  critical  to  success. 


Some  Limitations 

The  survey  questionnaire  process  used 
for  this  research  project  has  both  strengths 
and  weaknesses.  An  inherent  strength  of 
the  questionnaire  approach  lies  in  the  num¬ 
ber  of  program  managers  who  can  be  ac¬ 
cessed  simultaneously,  with  the  resultant 
savings  in  time  and  dollars.  It  provides  a 
neutral  and  standardized  method  of  data 
collection  and  allows  those  responding  to 
provide  not  only  the  identification  of 
CSFs,  but  also  a  view  of  the  assumptions 
underlying  the  CSFs  and  the  ways  in 

which  they  can 
be  measured. 

One  of  the 
weaknesses  of 
this  process  is 
the  lack  of  op¬ 
portunity  to  dis¬ 
cuss  a  question 
with  the  pro¬ 
gram  manager 
and  draw  out 
responses  that 
fully  consider  the  question  and  its  impli¬ 
cations.  This  can  be  very  important  when 
seeking  responses  from  managers  with 
regard  to  a  concept  or  process  they  are  not 
already  familiar  with.  It  is  subject  to  a  cer¬ 
tain  degree  of  bias  on  the  part  of  those 
responding  and  does  not  allow  for  pur¬ 
suit  of  additional  issues  that  might  be 
important  to  the  result  and  which  might 
not  be  explored  without  the  aid  of  a 
skilled  interviewer.  In  addition,  there  is 
no  protection  from  the  tendency  to  either 


Recommendations 


Based  on  the  information  provided 

above,  we  recommend  the  following. 

•  Educate  program  managers  and  their 
staff  in  the  CSF  identification  process. 
The  failure  to  explicitly  identify  CSFs 
for  a  program  will  invariably  result  in 
the  continued  focus  on  cost  and  sched¬ 
ule  after  they  become  problems,  will 
inhibit  the  development  and  use  of  ef¬ 
fective  life  cycle  measures,  and  will 
prevent  the  development  of  a  truly  ef¬ 
fective  program  management  informa¬ 
tion  system.  Cost  and  schedule  prob¬ 
lems  are  generally  effects,  not  causes. 
They  are  the  results  of  conditions  that 
should  be  identified  and  managed 
much  earlier  than  the  time  when  a  cost 
and  schedule  variance  first  appears. 

•  Educate  the  oversight  agencies  in  the 
CSF  identification  process  and  the  im¬ 
portance  of  this  to  their  management 
functions. 

•  Educate  the  program  managers  and 
their  staff  in  development  of  informa¬ 
tion  networks  consistent  with  the  criti¬ 
cal  success  factors. 

•  Establish  oversight  reporting  mecha¬ 
nisms  consistent  with  critical  success 
factors  so  that  critical  information  is 
reported  when  it  is  needed. 


"An  inherent 
strength  of  the 
questionnaire  ap¬ 
proach  lies  in  the 
number  of  program 
managers  who  can 
be  accessed  simulta¬ 
neously,  with  the 
resultant  savings  in 
time  and  dollars." 
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Continuing  Research 

Research  is  currently  being  conducted 
using  an  interview  process.  The  results 
will  reflect  whether  there  are  any  changes 
to  the  CSFs  reported  by  program  manag¬ 
ers  that  may  result  from  acquisition  reform 
policy  initiatives.  Continued  research  is 
also  being  conducted  to  produce  a  model 


for  evaluation  of  actual  criticality  of  re¬ 
ported  CSFs.  It  is  anticipated  that  the  de¬ 
velopment  of  this  model  will  provide  a 
means  to  alleviate  the  need  to  assume  that 
all  managers  are  equally  skilled  in  their 
ability  to  identify  CSFs  and  engage  in  ef¬ 
fective  strategic  thinking. 
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